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Project  Summary 

Recent  theoretical  developments  in  the  areas  of  latent  variable  modeling, 
random  coefficient  modeling,  multilevel  modeling,  missing  data  theory,  and 
empirical  Bayes  estimation  provide  interesting  research  opportunities  for  the 
psychometric  modeling  of  the  relationship  between  job  performance  and  test 
performance.  These  developments  suggest  new  approaches  to  the 
assessment  of  predictive  validity  of  tests  as  well  as  to  problems  of  selection 
and  assignment.  The  purpose  of  the  project  is  to  use  these  theories  as  a  basis 
for  developing  new  psychometric  methods  that  have  the  potential  for  better 
understanding  the  predictive  validity  of  tests  like  the  ASVAB  and  improving 
the  selection  and  assignment  of  recruits  for  military  jobs. 
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Progress  to  Date 


The  project  started  5/01/93.  This  starting  date  is  about  one  year  later  than 
what  was  proposed,  the  reason  being  lack  of  ONR-MPT  funding.  According 
to  the  proposal,  Jin-Wen  Yang  Hsu  was  to  work  with  the  PI  as  research 
associate  given  her  previous  experience  with  selection  modeling  in  a 
predictive  validity  context.  At  this  late  date,  however,  she  was  no  longer 
available  but  moved  to  Miami  to  start  an  assistant  professorship.  This 
change  has,  however,  not  hampered  the  woik  in  that  an  excellent  new  Ph.D. , 
Guanghan  Liu,  was  hired  in  June  after  he  received  his  Ph.D.  in  Statistics 
from  the  Mathematics  Department  at  UCLA.  Dr.  Liu  had  the  opportunity  to 
learn  relevant  techniques  from  Dr.  Yang  Hsu  before  she  left  UCLA. 

Project  activities  in  three  areas  will  be  reported:  gaining  access  to  suitable 
U.S.  military  data,  understanding  predictive  validity  from  a  latent  variable 
perspective,  and  Monte  Carlo  studies  of  a  conventional  method  for  assessing 
predictive  strength.  These  activities  are  in  line  with  the  original  research 
proposal. 

Gaining  access  to  U.S.  military  data 


As  discussed  in  the  research  proposal,  it  it  clear  that  analyses  of  U.S. 
military  data  relevant  to  assessing  predictive  validity  of  the  ASVAB  are 
valuable  both  to  illustrate  the  proposed  methodology  in  the  project  and  to 
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determine  which  further  methodology  development  was  needed.  As  soon  as 
the  project  started,  I  therefore  discussed  data  issues  with  the  contacts  I  had; 
Laurie  Wise  at  DMDC,  Anne  Mavor  at  NAS/NRC,  Jane  Arabian  at  the 
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Pentagon,  and  John  Wolfe  at  NPRDC.  It  may  be  of  interest  to  report  on  a 
summary  of  the  process  for  trying  to  get  data. 


As  a  number  of  the  NRC  conunittee  on  Military  Enlistment  Standards,  I  had 
learned  about  HumRRO's  work  on  the  Linkage  Project  using  the  JPM  hands- 
on  job  performance  measures.  I  thought  it  would  be  valuable  to  be  able  to 
use  these  data  to  enable  future  comparisons  of  their  linkage  equation  with 
the  prediction  equation  my  project  would  give.  This  led  to  a  specific  interest 
in  the  Army's  Project  A  data  and  the  nine  Batch  A  MOS  jobs,  although  data 
from  all  services  would  be  valuable.  I  contacted  Anne  Mavor  at  NRC  to  be 
able  to  gain  access  to  the  Project  A  data  via  HumRRO,  but  further 
discussions  with  Army  representatives  indicated  that  it  would  be  better  to 
obtain  the  data  via  them.  Jane  Arabian  discussed  my  interest  in  data  at  the 
July  1993  meeting  with  the  Manpower  Accession  Policy  Working  Group.  It 
became  clear  that  it  would  be  easiest  to  gain  access  to  data  from  Project  A, 
while  data  collected  by  the  other  services  may  take  longer  to  access,  mainly 
because  some  data  sets  were  still  being  analyzed  (such  as  the  ECAT  data) 
and  not  fully  explored  by  the  originators  of  the  data.  In  terms  of  the  Project 
A  data,  I  would  only  be  given  access  to  the  Concurrent  Validity  data  at  this 
point,  not  the  more  interesting  data  from  the  Longitudinal  Validity  Study.  I 
tried  to  reach  Major  Jim  Thomas  about  data  access  and  was  referred  to  Mr. 
Ron  Patsy  at  the  Headquarters  Department  of  the  Army.  A  data  request 
letter  was  sent  to  him  on  August  12  (the  data  request  letter  is  enclosed). 

He  agreed  to  have  Mike  Rumsey  at  ARI  help  me  widi  access  to  the 
Concurrent  Validity  data.  Rumsey  informed  me  that  data  could  be  produced 
by  AIR  and  referred  me  to  Gregoiy  Wilson  for  the  speciEc  data  layout. 

After  some  delays,  I  was  able  to  clarify  my  data  request  to  Wilson.  At  AIR, 
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I  also  talked  to  Norm  Peterson  who  was  very  helpful  in  terms  of  describing 
variables  collected.  It  was  finally  possible  on  October  6  to  send  a  purchase 
order  for  $250  to  cover  data  production  costs  for  the  nine  Batdi  A  MOS  jobs 
from  the  Concurrent  Validity  Study  from  UCLA  to  AIR.  The  data  tape  has 
not  yet  arrived. 

Understanding  Predictive  Validity  from  a  Latent  Variable  Perspective 

The  research  proposal  identified  seven  research  strands  to  be  studied  in  this 
project.  Work  has  begun  on  two  of  those  strands.  Within  the  first  strand. 
Predictive  Validity  from  a  Latent  Variable  Perspective,  software  has  been 
written  for  investigating  differences  in  predictive  validity  assessment  using 
three  different  methods:  regression  analysis  with  observed  test  scores  as 
predictors;  regression  analysis  with  factor  scores  as  predictors;  and  structural 
equation  modeling  with  latent  variables.  With  a  given  latent  variable  model 
for  a  set  of  test  scores,  it  is  asked  how  the  different  methods  represent 
incremental  validity  contributed  by  added  subtests.  This  is  a  typical  question 
asked  in  the  context  of  proposed  additions  to  ASVAB  in  terms  of  new 
cognitive  predictors.  Questions  of  significance  of  predictor  composites  are 
also  considered.  The  project  currently  investigates  what  advantage  latent 
variable  models  have,  if  any,  for  identifying  and  correctly  estimating  the 
predictive  strength  of  minor  factors  operating  in  a  test  battery.  Initial  work 
indicates  that  the  predictive  strengths  of  neither  minor  nor  major  factors  are 
correctly  described  when  latent  variable  methods  are  not  used.  Implications 
for  selection  in  terms  of  composites  of  observed  test  scores  are  being 
investigated  (this  also  leads  to  research  strand  5:  Issues  of  Selection  and 
Assignment  by  Latent  Variable  Models). 
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Monte  Carlo  studies  of  a  conventional  method  for  assessing  predictive 
strength 

Another  research  strand  of  the  proposal  is  labelled  Issues  of  Selective 
Samples.  Here,  the  key  issue  of  correctly  assessing  predictive  validity  from 
the  sample  of  individuals  selected  into  a  training  program  or  a  job  is 
considered.  Several  different  techniques  for  drawing  inference  to  the 
population  of  applicants  are  to  be  studied  in  the  project.  It  was  decided  that 
an  important  starting  point  would  be  to  consider  the  most  commonly 
employed  approach  to  this  problem.  This  method  uses  Pearson-  Lawley 
corrections  (adjustments  for  restriction  of  range).  In  the  context  of  latent 
variable  modeling,  an  adjustment  is  made  to  the  covariance  matrix  for  die 
selected  sample  and  the  latent  variable  analysis  carried  out  on  this  matrix. 
This  appears  to  be  one  of  the  methods  considered  in  the  ECAT  study. 
Software  has  been  written  for  Monte  Carlo  studies  of  the  sampling  behavior 
of  this  common  estimator.  It  appears  not  to  have  been  rigorously  studied 
previously.  The  performance  of  this  estimator  is  now  being  compared  to 
that  of  the  factor  score  estimator  and  the  theoretically-optimal  maximum 
likelihood  estimator  which  have  been  studied  to  a  certain  extent  in  pilot 
woik  for  the  project  (for  background,  see  the  original  proposal). 
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